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The  Differential  Classification  Work  Unit  of  the  Behavior  and  Systems  Research 
Laboistory  has  as  a  primary  objective  the  continuing  cono«.jt  of  research  to  maintain 
and  Improve  the  effectiveness  of  the  Army  Classification  Battery  (ACB).  As  a  part  of 
the  cvera1!  effort,  the  Work  Unit  provides  assistance  to  Army  operational  agencies  in 
developing  efficient  scoring,  reporting,  and  utilization  techniques.  The  present  Tech¬ 
nical  Research  Note  deals  with  the  problems  of  attrition  at  the  Army  Ordnance  School 
and  reports  on  a  segment  of  a  broader  research  effort  to  find  ways  of  reducing  attri¬ 
tion  among  ordnance  trainees. 

Scores  on  the  ACB  predictors  and  written  and  performance  tests  given  during  three 
Ordnance  courses  were  subjected  to  analyses  to  attempt  to  account  for  failure  to  com¬ 
plete  training  satisfactorily.  Specific  objectives  of  the  present  analysis  were  to 
determine  l)  failure  rates  during  the  course,  2)  consistency  of  grades  at  different 
reporting  periods,  and  3)  consistency  of  measured  aptitudes  and  course  performance.  The 
three  courses  selected  (because  of  high  failure  rates)  for  analysis  were:  Machinist 
(44E),  Small  Arms  Repair  (45B),  and  Fuel  and  Electrical  System  Repair  (63C).  Scores 
on  written  and  performance  tests  given  at  the  end  of  each  reporting  period  in  the 
courses  were  analyzed  in  relation  to  each  other,  to  final  course  grade,  and  to  ACB  test 
scores.  Failure  rates  on  the  test  during  and  at  end  of  course  were  ascertained. 

Results  of  the  analyses  showed  that  ACB  tests  had  the  expected  degree  of  effective¬ 
ness  as  predictors  of  trainee  performance  (correlation  coefficients  in  the  .50's  and 
.Go's).  Failure  rates  in  most  written  tests  were  high,  sometimes  exceeding  50%  of 
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those  completing  the  course.  Failure  rates  on  performance  tests  were 
low — under  — except  In  the  Machinist  course  (over  20$  level).  Obtain¬ 

ed  Interrelationships  between  written  and  performance  testa  were  Incon¬ 
sistent  in  respect  to  individuals  passing  and  falling,  as  was  also  the 
case  for  failure  rates  on  tests  at  early  and  late  reporting  periods. 


Reevaluation  of  written  testa  given  at  the  end  of  each  period  may 
be  of  greater  utility  in  reducing  attrition.  A  subsequent  research  phase 
will  be  directed  to  examining  the  relationships  among  aptitude  tests, 
training  grades,  and  performance  on  job  tasks  in  an  effort  to  gain 
understanding  of  how  these  measures  function  in  the  total  process  of 
classifying,  training,  and  utilizing  Army  enlisted  men. 
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FOREWORD 


The  DIFFERENTIAL  CLASSIFICATION  Work  Unit  applies  psychological  measurement 
methods  to  enable  the  Army  to  make  best  use  of  the  different  skills  and  aptitudes  of  its  en¬ 
listed  personnel  through  increasingly  accurate  and  differentiated  measures  of  individual  poten¬ 
tial.  Research  is  conducted  to  maintain  and  improve  the  effectiveness  of  the  Armv  Classification 
Battery  and  related  techniques  and  to  determine  how  different  conditions -changes  in  training 
programs  and  job  content  and  environment,  for  exempt-may  interact  with  classification  test 
measures  and  thus  affect  the  basis  for  utilization  ot  the  enlisted  input. 

As  a  part  of  i  „  overall  effort,  the  Work  Unit  provides  assistance  to  the  Army's  operational 
agencies  in  developing  efficient  scoring,  reporting,  and  utilization  technique'.  Recommendations 
are  based  in  large  part  on  research  and  systems  analysis  of  the  entire  process  of  classifying,  train¬ 
ing,  and  utilizing  enlisted  men  in  the  various  Military  Occupational  Specialties. 

The  present  Technical  Research  Note  deals  with  problems  of  attrition  at  the  Army  Ord¬ 
nance  School,  and  presents  an  analysis  of  scores  on  Army  Classification  Battery  predictors  and 
written  and  performance  tests  given  during  three  Ordnance  courses  in  an  effort  to  account  for 
failure  to  complete  the  training  satisfactorily. 

The  entire  research  work  unit  is  responsive  to  special  requirements  of  the  Deputy  Chief 
of  Staff  for  Personnel  and  the  U.  S.  Continental  Army  Command,  as  well  as  to  objectives  of 
Army  RDT&E  Project  2Q062106A722,  "Selection  and  Behavioral  Evaluation,"  FY  1972  Work 
Program. 
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BRIEF 


Requirement: 

To  analyze  a  sequence  of  grades  on  tests  in  Ordnance  school  courses  in  relation  to  failure 
rates  and  Army  Classification  Battery  test  scores  as  a  segment  of  research  to  reduce  attrition 
among  Ordnance  trainees. 


Procedure: 

Scores  on  v\,  itten  and  performance  tests  given  at  the  end  of  each  reporting  period  in  three 
Ordnance  school  courses- Machinist  (44E),  Smalt  Arms  Repair  (15B),  and  Fuel  and  Electrical 
System  Repair  (63G)-were  analyzed  in  relation  to  each  other,  to  final  course  grade,  and  to  ACB 
test  scores.  Failure  rates  on  written  and  performance  tests  during  and  at  end  of  course  were 
ascertained. 


Findings: 

ACB  tests  had  the  expected  degree  of  effectiveness  in  predicting  trainee  performance  (corre¬ 
lation  coefficients  in  the  ,50's  and  ,60's).  Tests  in  the  aptitude  area  prerequisites  for  the  courses 
were  the  best  predictors  for  the  appropriate  courses. 

Failure  rates  in  most  written  tests  were  high,  sometimes  exceeding  50  percent  of  those 
completing  the  course. 

On  performance  tests,  failure  rates  were  low,  under  5  percent  except  In  the  Machinist 
course  in  which  over  20  percent  failed  some  tests. 

Written  and  performance  tests  were  inconsistent  in  respect  to  individuals  passing  and  failing. 

•  » 

Failure  rates  on  tests  at  early  and  late  reporting  periods  were  also  inconsistent. 


Utilization  of  Findings: 

The  generally  constant  effectiveness  of  the  ACB  tests  supports  their  continued  use  as  prereq¬ 
uisites.  Current  aptitude  areas  for  each  course  are  appropriate. 

However,  raising  the  prerequisite  score  would  lower  attrition  only  slightly  and  would  reduce 
the  eligible  manpower  pool.  A  more  fruitful  approach  to  reducing  attrition  is  to  reevaluate  the 
written  tests  given  at  the  end  of  each  reporting  period.  These  tests,  which  are  a  prime  determiner 
of  pass-fail,  may  cover  more  difficult  skills  and  concepts  than  are  required  at  the  entry  level. 
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THE  PROBLEM 

The  present  study  was  conducted  as  part  of  a  broader  research  effort 
to  find  ways  of  reducing  attrition  in  Ordnance  School  courses.  In  thia 
effort,  attrition  Is  viewed  not  as  an  isolated  phenomenon  but  as  one  as¬ 
pect  of  the  entire  process  of  classification,  training,  and  utilization 
of  enlisted  men.  For  this  reason,  the  research  deals  with  the  inter¬ 
relationships  of  aptitude  tests,  training  grades,  and  performance  on  the 
Job. 


Basic  to  an  understanding  of  the  process  of  assigning  men  to  train¬ 
ing  is  information  on  the  predictive  accuracy  of  the  aptitude  tests  and 
on  the  interrelationships  of  the  achievement  measures  entering  into  eval¬ 
uations  of  performance  in  the  training  course  (including  failure  to  com¬ 
plete  the  course  satisfactorily).  Components  of  course  grade  Include 
written  and  performance  tests  given  during  and  at  the  end  of  the  course. 
The  present  Technical  Research  Note  231  covers  an  analysis  of  the  inter¬ 
actions  between  Army  Classification  Battery  test  scores  and  the  written 
and  performance  tests  given  at  successive  reporting  periods  in  several 
Ordnance  courses.  The  aim  is  to  account  for  failure  to  complete  the 
training  satisfactorily. 

An  Important  function  of  achievement  tests  given  during  training  is 
to  determine  which  students  have  developed  an  adequate  level  of  skills 
and  knowledge.  Achievement  tests  are  also  used  to  make  distinctions 
among  those  passlng--for  example,  to  assign  grades  of  A,  B,  or  C  or  to 
identify  top  students  in  a  class.  The  measurement  instruments  used  are 
critical  in  assessing  the  students.  The  tests  must  be  relevant  to  the 
material  taught  (a  matter  of  expert  judgment)  and  they  must  make  adequate 
and  reasonable  discriminations  among  different  levels  of  achievement.  To 
make  an  intelligent  judgment  about  the  adequacy  and  reasonableness  of 
tests  applied  in  a  course,  an  analysis  of  the  scores  is  required.  In  the 
present  phase  of  the  research,  the  achievement  test  sc ores --writ ten  and 
performance--obtalned  at  the  end  of  each  reporting  period  in  three  Job 
training  courses  were  analyzed  in  relation  to  each  other,  to  final  course 
grade,  and  to  aptitude  test  scores  obtained  prior  to  training.  The 
research  was  designed  to  provide  data  about  the.  ways  the  tests  are  actu¬ 
ally  functioning.  The  results  can  be  evaluated  in  the  context  of  the 
instructional  model  on  which  the  courses  are  based.  The  question  is 
whether  the  results  are  in  keeping  with  the  objectives  that  guided  the 
development  of  the  courses  or  whether  there  is  some  dissonance  between 
observed  and  expected  outcomes  of  training. 


I 


OBJECTIVES 


The  specific  objectives  of  the  present  analysis  were  to  determine 
l)  failure  rates  during  the  course,  2)  consistency  of  grades  at  different 
reporting  periods,  and  3)  consistency  of  measured  aptitudes  and  course 
performance.  Failure  rates  during  instruction  were  examined  separately 
for  written  and  performance  tests  and  for  each  reporting  period  and  at 
the  end  of  the  course.  The  analysis  was  also  designed  to  show  l)  whether 
written  and  performance  tests  provide  similar  Information  about  the  skills 
and  knowledge  of  the  trainees,  2)  whether  the  same  or  different  trainees 
tend  to  do  well  during  the  different  reporting  periods,  and  3)  whether  the 
aptitudes  required  are  relevant  to  success  in  the  various  stages  of  lnstruc 
tlon. 


METHOD 

Courses  were  selected  for  analysis  because  they  had  high  failure 
rates,  input  was  large  enough  to  provide  an  adequate  sample,  and  course 
content  had  not  been  recently  revised  and  was  not  expected  to  be  changed 
in  the  near  future.  The  three  courses  selected  were: 

Machinist,  44E 

Small  Arms  Repair,  45B 

Fuel  and  Electrical  System  Repair,  63G 

The  samples  consisted  of  men  who  graduated  in  January  through  December 
1969,  and  included  only  inductees  and  enlistees  who  had  recently  begun 
their  tour  of  duty.  Reservists  on  active  duty  for  six  months  and  men 
who  had  had  other  extensive  Army  training  and  experience  were  excluded 
If  they  could  be  identified.  The  results,  then,  apply  to  new  Army  re¬ 
cruits  assigned  to  their  Initial  job  training  course;  attrition  is  more 
of  a  problem  with  recruits  than  with  reservists  on  active  duty. 

Academic  failure  rates  provided  by  the  Ordnance  School  are  shown 
below.  These  figures  are  based  on  the  entire  input,  Including  reservists. 

Course 

FUEL  AND  ELECTRICAL 


YEAR 

MACHINIST 

(44E) 

SMALL  ARMS  REPAIR 

(45B) 

SYSTEM  REPAIR 
(636) 

1968 

11* 

12* 

8* 

1969 

8* 

6* 

4% 

1970 

12* 

5* 

2* 

The  fluctuating  failure  rates  suggest  that  results  might  be  differ¬ 
ent  for  different  time  periods.  The  samples  for  each  course  were  divided 
Into  early  and  late  halves  and  each  half  was  analyzed  separately. 


The  Machinist  course  had  nine  reporting  periods,  with  nine  written 
and  six  performance  tests.  The  Small  Arms  Repair  course  also  had  nine  ' 
reporting  periodo  with  nine  written  and  four  performance  cests.  The.  Fuel 
and  Electrical  System  Repair  course  had  two  grading  plans,  one  used  until 
March  1969  (called  6?G-01d  in  the  accompanying  tables)  and  the  second  after 
that  (called  63G-New) .  The  earlier  plan  had  12  reporting  periods,  with 
12  written  and  six  performance  tests.  In  all  courses,  the  performance 
tests  were  more  heavily  weighted  in  the  final  course  grade  than  the  writ¬ 
ten  tests.  An  exception  is  the  earlier  version  of  the  Fuel  and  Electrical 
System  Repair  course  where  the  written  examinations  and  quizzes  carried 
57  percent  of  the  weight  in  the  final  grade;  in  the  revised  grading  plan, 
the  written  portions  had  only  38  percent  of  the  weight.  Grading  plans  for 
the  courses  are  presented  in  Appendix  A. 

Mental  characteristics  were  measured  by  the  Army  Classification  Bat¬ 
tery  (ACB)  and  the  Armed  Forces  Qualification  Test  (AFQT).  The  ACB  pro-  , 
vides  comprehensive  measurement  of  differential  aptitudes;  the  tests  are 
described  in  Table  1.  The  AFQT  is  a  measure  of  general  mental  ability 
used  as  an  initial  screen  to  determine  qualification  for  military  service. 

I 

RESULTS 

i 

Observed  failure  rates  in  these  samples  were  29  percent  for  the 
Machinist  course,  20  percent  for  Small  Arms  Repair,  and  11  percent  and 
18  percent  for  the  old  and  new  versions  of  the  Fuel  and  Electrical  Sys¬ 
tem  Repair  course,  respectively.  These  figures  are  higher  than  those 
reported  by  the  school  for  academic  reasons  only.  i 

I 

In  determining  which  reporting  periods  made  the  largest  contribution 
to  the  failure  rates,  all  individuals  failed  during  the  course  were  omitted 
from  the  sample  to  keep  the  basis  of  comparison  constant.  Qnly  those  men 
completing  the  course  were  used  in  measuring  the  difficulty  of  the  tests. 
Some  of  these  men  were  failed  at  the  end  of  the  course,  but  they  did  fin-  i 
ish  the  course  and  scores  on  all  tests  were  available  for  them.  Diffi¬ 
culty  of  the  testa  was  determined  by  finding  the  percentage  of  men  who 
failed  each  one. 

Results  for  the  Machinist  course  are  presented  in  Figure  1.  The  first 
two  written  tests  and  the  first  performance  test  given  during  the  course 
proved  very  easy,  with  less  than  10  percent  of  the  trainees  in  the  sample 
receiving  a  falling  grade.  The  other  tests  were  harder,  except  the  final 
written  test,  which  was  failed  by  slightly  under  10  percent.  The  final 
performance  test,  though,  was  difficult  with  about  26  percent  failing.  : 


> 


Table  1 


TESTS  IN  THE  ARMY  CLASSIFICATION  BATTERY 


1.  Verbal  Teat,  VE  (50  items).  Each  item  requires  the  examinee 
to  select  the  correct  synonym  for  the  underlined  word  in  a 
short  sentence. 

2.  Arithmetic  Reasoning,  AR  (40  items).  Each  item  is  a  reasoning 
problem  Involving  application  of  arithmetic  processes. 

3.  Pattern  Analysis,  PA  ( 50  items).  A  two-dimensional  pattern 
with  numbered  lines  is  presented  along  with  the  corresponding 
three-dimensional  figure  made  by  folding  the  pattern  along 
the  indicated  lines.  The  examinee  is  required  to  identify  the 
lettered  edge  of  the  figure  corresponding  to  a  numbered  line 
in  the  pattern. 

4.  Mechanical  Aptitude,  MA  (45  items).  Each  item  includes  a  fig¬ 
ure  illustrating  some  physical  principle. 

5.  Army  Clerical  Speed,  ACS.  In  Part  I,  Number  Reversal  (60 
items) .  the  examinee  Indicates  whether  the  second  number  in 
each  item  is  exactly  the  reverse  of  the  first.  In  Part  II. 
Coding  (50  items),  a  key  word  is  followed  by  a  number  that 
is  associated  with  it.  Each  item  presents  a  word  followed 
by  all  numbers  in  the  key.  The  examinee  is  to  pick  the  num¬ 
ber  corresponding  to  the  word  in  the  key. 

6.  Army  Radio  Code,  ARC — an  auditory  test,  recorded  on  tape.  The 
examinee  is  taught  the  code  signals  for  three  letters  I,  N,  and 
T.  Immediately  after  the  learning  exercises,  a  test  of  150 
iteu  is  given. 

7.  i  Shop  Mechanics,  SM  (40  items).  Each  item  presents  a  drawing 

Illustrating  some  mechanical  principle  or  tool  usage. 

8.  Automotive  Information,  AI  ( 40  items).  Each  item  is  a  question 
about  the  identification  or  operation  of  automobile  parts. 

9.  Electronics  Information,  ELI  (40  items).  The  examinee  is  re¬ 
quired  to,  associate  pictured  objects  in  terms  of  how  they 
function  electronically,  and  in  verbal  items  to  demonstrate 
his  knowledge  of  electronics  principles. 

10. 1  Classification  Inventory,  Cl  (125  items).  The  Cl  consists  of 
self-description  items  in  which  the  examinee  indicates  his 
personal  background,  attitudes,  self-evaluation,  and  experi¬ 
ences. 

11.  Geiieral  Information  Test,  GIT  ( 50  items).  Questions  cover 
objective  items  of  information  about  various  avocational 
pursuits. 
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If  the  minimum  passing  grade  is  designed  to  assess  the  standards  that 
every  trainee  should  attain,  then  either  most  of  the  later  tests  were  too 
difficult  or  the  instruction  did  not  bring  a  large  percentage  of  trelnees 
up  to  standard.  Since  failure  rates  on  all  written  and  performance  tests 
were  based  on  the  same  men,  the  failure  rates  were  primarily  a  function  of 
the  tests  themselves  and  not  of  different  men  taking  the  tests.  Of  course, 
performance  of  individuals  does  vary  across  time  and  may  have  had  a  slight 
effect  on  failure  rates.  The  test  means  for  all  courses  are  presented  in 
Appendix  B. 

The  percentage  of  failures  in  the  Small  Arms  Repair  course  (45B)  showed 
a  different  pattern  than  in  the  Machinist  course  (Figure  2).  The  earlier 
written  tests  were  difficult,  with  36  percent  failing,  In  the  sixth  and 
seventh  reporting  periods,  failure  rate  dipped  below  10  percent.  The  final 
written  examination  was  extremely  difficult,  with  60  percent  failing.  Note 
that  all  the  trainees  falling  the  final  completed  the  course,  although 
not  all  were  successful.  The  performance  tests  were  considerably  easier, 
with  a  maximum  of  18  percent  failing  the  performance  test  for  period  6. 

Most  falling  scores  In  the  Small  Arms  Repair  course  occurred  on  written 
tests,  relatively  few  on  performance  tests. 

The  failure  rates  In  both  versions  of  the  Fuel  and  Electrical  System 
Repair  course  also  showed  large  differences  between  written  and  performance 
tests.  Tests  for  Course  fi3G-01d,  shown  in  Figure  3,  started  easy,  and 
then  became  more  difficult  for  the  trainees  until  period  8,  when  failure 
rate  fell  to  about  20  percent.  Again,  the  final  written  test  was  diffi¬ 
cult.  Course  63G-New  started  difficult  and  ended  even  more  so,  as  shown 
lr,  Figure  4.  About  half  failed  the  first  written  test  and  85  percent 
railed  the  final  one.  The  performance  tests,  in  contrast,  had  almost 
no  failures.  In  the  new  course,  only  the  performance  test  for  period 
6  was  failed;  no  one  failed  the  performance  tests  for  periods  1,  3, 

7,  and  8. 

Analysis  of  failure  rates  for  the  written  and  performance  tests  indi¬ 
cated  that  written  tests  were  more  difficult  than  performance  tests  in 
the  Small  Arms  Repair  and  Fuel  and  Electrical  System  Repair  courses, 
while  in  the  Machinist  course  written  and  performance  tests  were  of  more 
equal  difficulty.  The  performance  tests  are  so  heavily  weighted  in 
computing  final  course  grade  that  the  overall  failure  rates  are  largely 
a  function  of  performance  tests.  In  Course  63G-New,  the  failure  rate 
was  only  18  percent  even  though  83  percent  failed  the  final  written  test. 

As  shown  in  Appendix  A,  27  percent  of  the  weight  for  the  final  grade  in 
this  course  is  assigned  to  the  written  examinations,  while  62  percent 
is  assigned  to  the  performance  tests.  The  other  11  percent  goes  to  graded 
quizzes.  The  effect  was  to  assign  a  high  constant  to  each  trainee  in 
63G-New  based  on  his  performance  scores.  Variation  between  trainees  was 
thus  largely  a  function  of  the  written  tests.  The  same  effect  was  noted 
for  the  Small  Arms  Repair  course,  but  not  for  the  Machinist  course.  In 
the  Machinist  course,  both  the  overall  written  and  performance  failure 
rates  were  high,  and  the  performance  tests  did  discriminate  between 
failures  and  passers. 


Figure  1.  Fai 


The  proportion  of  failures  provides  information  on  how  difficult  the 
test  is,  but  the  question  still  remained  whether  the  same  or  different 
men  tended  to  fall  successive  tests.  The  second  objective  of  the  anal¬ 
ysis  was  to  determine  whether  trainees  tended  to  perform  at  a  constant 
level  or  whether  Individuals'  grades  fluctuated  from  reporting  period  to 
reporting  period.  Another  aspect  was  to  determine  whether  trainees  with 
high  aptitude  tended  to  do  well  in  all  reporting  periods. 

Analysis  of  consistency  of  performance  focused  on  the  training  program 
Itself:  How  do  grades  attained  early  in  the  course  compare  with  those 
later  on?  How  do  grades  on  written  tests  compare  with  those  on  perfor¬ 
mance  tests?  Is  there  a  shift  in  the  relationships  as  training  progresses? 

The  relationships  between  grades  on  the  first  and  last  written  tests 
are  presented  in  Table  2.  The  last  test  is  defined  as  the  one  just  pre¬ 
ceding  the  end-of-courte  test.  The  final  test  covers  all  material  from 
all  reporting  periods,  while  the  last  test  covers  only  the  material 
taught  in  the  period  immediately  before  the  final.  For  example,  the  last 
test  in  the  Machinist  course  covers  reporting  period  8,  while  the  final 
test  covers  reporting  period  9  and  all  preceding  periods.  Of  the  350 
trainees  who  completed  the  course,  79  failed  the  last  written  test,  while 
only  9  failed  the  first.  Six  of  these  9  also  failed  the  last.  Fifteen 
trainees  scored  85-IOO  on  the  first  test  but  failed  the  last.  The  figures 
shown  for  this  course  reflect  a  correlation  coefficient  of  .51.  The  grades 
for  the  Small  Arms  Repair  and  the  Fuel  and  Electrical  System  Repair  course 
were  considerably  less  consistent,  as  indicated  by  correlation  coefficients 
of  .29  and  .30,  respectively.  In  the  Small  Arms  Repair  course,  12  of 
265  trainees  scored  85-IOO  on  the  first  test  and  failed  the  last.  In  the 
old  form  of  the  Fuel  and  Electrical  System  Repair  course,  failures  on 
the  first  written  test  were  not  counted  separately  since  there  were  only 
three.  In  the  later  form,  the  last  written  test  was  extremely  difficult, 
with  only  46  of  186  scoring  67  and  above.  In  all  the  courses,  there 
was  considerable  variation  in  performance  between  first  and  last  written 
tests. 

Consistency  of  grades  on  performance  tests  is  shown  In  Table  3»  The 
relationship  was  lower  for  performance  tests  than  for  written  tests.  Of 
the  32  men  in  the  Machinist  course  who  failed  the  last  performance  test, 
only  eight  scored  below  79  on  the  first,  while  the  remaining  24  scored 
80  or  above.  In  the  Small  Arms  Repair  course  and  the  older  form  of  the 
Fuel  and  Electrical  System  Repair  course,  a  similar  degree  of  relationship 
was  found.  In  the  new  course,  the  two  measures  were  statistically  inde¬ 
pendent,  as  shown  by  the  correlation  coefficient  of  .01.  The  relative 
standing  of  the  trainees  on  performance  tests  changed  considerably  between 
first  and  last  testing  periods.  The  mean  coefficients  presented  in 
Append!  r  show  that  those  presented  in  Tables  2  and  3  are  not  atypical. 
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Figure  2.  Failure  rate  at  each  reporting  period  in  the  Small  Arms  Repair  Course  (45B) 


RELATIONSHIPS  BETWEEN  FIRST  AND  LAST  WRITTEN  TESTS 
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Another  measure  of  consistency  of  test  scores  is  provided  by  a  compar¬ 
ison  of  the  final  written  and  performance  tests  (Table  4).  Since  the  final 
tests  covered  the  entire  course,  their  relationship  with  the  other  tests 
tended  to  be  higher  than  that  for  tests  limited  to  a  single  reporting 
period.  The  low  correlation  shown  in  Table  4  indicates  that  in  the  Ma¬ 
chinist  and  Small  Arms  Repair  courses  final  written  and  performance  scores 
were  quite  Independent.  In  both  forms  of  the  Fuel  and  Electrical  System 
Repair  course,  however,  written  and  performance  tests  did  have  a  moderate 
relationship.  In  the  Machinist  course,  only  17  of  329  failed  the  final 
written  test,  while  8l  failed  the  final  performance  test.  Of  the  8l  fail¬ 
ures,  27  scored  85-IOO  on  the  final  written  test  and  only  8  failed  it. 
Thus,  the  two  final  written  and  performance  tests  provided  discrepant  in¬ 
formation  about  how  capable  the  trainees  were  at  the  end  of  the  course. 

When  correlation  coefficients  between  written  and  performance  tests 
for  the  same  reporting  periods  were  examined,  the  same  low  consistency 
was  found  for  most  periods.  The  mean  correlation  coefficients  are  pre¬ 
sented  in  Appendix  C. 

Scores  tended  to  fluctuate  across  time,  as  revealed  in  the  analysis 
of  the  test  scores  for  the  separate  reporting  periods.  Performance  test 
scores  were  less  consistent  than  written  test  scores.  Most  tests --written 
and  performance --had  high  correlation  with  final  course  grades,  the  bulk 
of  the  coefficients  being  above  .60  (Appendix  c) .  One  reason  for  the 
generally  high  correlation  with  final  course  grade  was  that  each  test  was 
a  contributor  to  the  final  grade  and  thus  there  was  a  part -whole  relation¬ 
ship.  Another  reason  was  that  almost  all  the  intercorrelations  were 
positive  and  there  was  some  cumulative  effect  of  the  test  scores,  even 
if  the  tests  for  some  reporting  periods  were  relatively  independent  of 
those  for  other  periods. 

The  next  step  in  analyzing  consistency  of  performance  was  to  examine 
the  relationship  between  ACB  aptitude  tests  administered  prior  to  train¬ 
ing  and  achievement  tests  administered  during  training.  The  mean  corre¬ 
lation  coefficients  of  ACB  with  training  period  grades  are  presented  in 
Appendix  C;  selected  coefficients  are  discussed  in  the  text.  All  coeffi¬ 
cients  have  been  statistically  corrected,  via  the  multivariate  restriction 
model,  to  be  estimates  of  the  values  that  would  be  found  for  a  representa¬ 
tive  uneelected  sample  from  the  mobilization  population. 

Two  assumptions  made  in  interpreting  correlation  coefficients  as 
showing  the  predictive  accuracy  of  an  aptitude  test  score  are:  l)  that 
the  relationship  between  the  predictor  (ACB  tests)  and  criterion  (training 
grades)  is  essentially  linear  throughout  the  useful  score  range,  and  2) 
that  the  errors  of  prediction  have  the  same  standard  deviation  for  all 
levels  of  predictor  scores.  Scatterplovs  of  aptitude  scores  with  final 
course  grade  for  each  course  revealed  that  the  two  assumptions  were  aet 
in  these  data.  The  correlation  coefficients  can  therefore  be  Interpreted 
as  reflecting  the  true  degree  of  predlccive  validity  of  the  ACB  aptitude 
tests . 


RELATIONSHIP  BETWEEN  FINAL  WRITTEN  AND  PERFORMANCE  TESTS 
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The  validity  of  the  ACB  in  predicting  course  achievement  wee  also 
determined  separately  for  classes  beginning  early  in  the  sampling  period 
and  those  beginning  later.  The  validity  of  the  ACB  was  found  to  be  lower 
for  the  later  classes  in  the  Machinist  course,  and  the  two  seta  of  results 
are  presented  for  this  course.  No  differences  were  found  for  the  Small 
Arms  Repair  course,  and  only  the  results  for  the  total  sample  are' pre¬ 
sented.  The  Old  and  New  Fuel  and  Electrical  System  Repair  courses  were 
analyzed  and  are  reported  separately. 

The  validity  of  the  tests  in  the  aptitude  area  scores  used  as  prereq¬ 
uisites  is  shown  in  Table  5*  Also  shown  is  the  validity  of  the  General 
Information  Test  (GIT),  which  measures  both  general  mental  ability  and 
mechanical  ability.  The  tests  in  the  aptitude  area  prerequisites  for 
given  courses  had  higher  validity  coefficients  than  the  GIT  for  all 
courses  except  the  more  recent  form  of  the  Fuel  and  Electrical  System 
Repair  course  where  validity  was  about  equal.  The  validity  of  the  Pattern 
Analysis  and  Shop  Mechanics  tests,  which  make  up  the  General  Maintenance 
Aptitude  Area,  declined  for  the  later  classes  in  the  Machinist  course. 

The  later  classes  graduated  in  the  latter  part  of  1969,  most  of  the  ear¬ 
lier  classes  early  in  1969*  The  validity  of  the  General  Information 
Test  showed  similar  decline.  Most  of  the  other  ACB  tests  also  had  lower 
validity  for  the  later  classes  in  the  Machinist  course.  Apparently,  some 
change  occurred  in  this  course  that  had  the  effect  of  reducing  the  predic¬ 
tive  accuracy  of  the  ACB  tests. 

The  matrices  were  examined  to  determine  whether  meaningful  patterns 
of  correlation  emerged.  One  facet  examined  was  whether  tests  appropriate 
to  a  given  course  had  the  higher  coefficients.  In  all  cases,  the  tests 
in  the  aptitude  area  prerequisites  for  the  courses  had  highest  or  close 
to  highest  validity.  The  tests  measuring  mechanical  ability,  such  as  the 
Mechanical  Aptitude  and  Automotive  Information  tests,  had  higher  validity 
than  those  of  general  mental  ability,  such  as  Verbal. 

In  respect  to  the  validity  profile  of  the  tests  for  successive  report¬ 
ing  periods,  one  hypothesis  is  that  tests  are  valid  for  the  initial  periods 
of  instruction,  but  then  lose  their  validity  as  the  training  material 
begins  to  build  on  what  was  taught  previously.  An  alternative  hypothesis 
is  that  the  validity  of  a  test  incres.ss  for  later  periods  of  instruction: 
If  each  unit  builds  on  previous  units  in  a  cumulative  manner  and  the  able 
trainees  master  the  successive  steps  while  the  dull  ones  fall  progressively 
further  behind,  then  the  validity  of  relevant  tests  would  increase.  The 
validity  profiles  for  the  Machinist  and  Fuel  and  Electrical  System  Repair 
courses  were  generally  flat,  meaning  that  most  ACB  tests  had  relatively 
constant  validity  across  time  for  these  two  courses.  In  these  two  courses, 
neither  hypothesis  about  decreasing  or  increasing  validity  was  supported. 

In  the  Small  Arms  Repair  course,  a  consistent  pattern  of  decreasing 
validity  did  emerge  for  the  first  eight  written  tests.  The  validity 
increased  again  for  the  final  end-of-course  written  test.  The  same 
decreasing  pattern  emerged  slightly  when  the  performance  tests  were 
considered.  It  is  possible  that  in  the  Small  Arms  Repair  course  the 


Table  5 

VALIDITY  OF  SELECTED  ACB  TESTS 


Tests 


General 

Maintenance 

Aptitude  Area 

Course 

PA 

SM 

GIT 

Mach  ini  s  t  -  -E  ar  ly 

.62 

•  47 

Machinist --Late 

•59 

.42 

•15 

Small  Arms  Repair-Total 

.58 

•53 

•50 

Motor 

Maintenance 

Aptitude  Area 

MA 

AI 

Fuel  and  Electrical-Old 

•  71 

.68 

.64 

System  Repair-New 

.54 

•57 

.60 

nature  of  the  instruction  was  such  that  learning  the  later  materials  re¬ 
quired  less  of  the  aptitudes  measured  by  the  ACB.  An  examination  of  the 
curriculum  would  be  required  to  support  a  hypothesis  about  the  progression 
of  learning. 

The  generally  constant  validity  of  the  ACB  tests  supports  their  use 
as  prerequisites.  The  appropriate  tests  were  able  to  identify  men  who 
are  likely  to  be  successful.  The  tests  were  about  equally  effective 
throughout  the  course  and  for  predicting  final  course  grade.  Since  the 
tests  generally  had  consistent  predictive  validity,  one  possibility  for 
reducing  failure  rates  is  to  Increase  the  prerequisite  score.  The  data 
were  examined  to  determine  the  percentage  of  failures  at  different  score 
levels  on  the  aptitude  area  prerequisites.  The  results  are  presented  in 
Table  6.  In  all  three  courses,  some  trainees  were  accepted  who  did  not 
meet  the  established  prerequisites.  In  all  cases  except  the  earlier  form 
of  the  Fuel  and  Electrical  System  Repair  course,  the  failure  rate  of  these 
underqualified  men  was  high,  ranging  from  J>8  percent  to  44  percent.  The 
total  failure  rate  in  the  course  was  low,  however,  (6$),  and  there  was  no 
consistent  drop  in  failures  as  prerequisite  scores  increased.  In  the 
Machinist  course,  the  total  failure  rate  was  high,  16  percent,  and  in  the 
GM  score  interval  II5-II9  the  failure  rate  was  still  10  percent. 

While  raising  the  prerequisite  score  would  lower  attrition  in  all 
courses,  many  men  who  successfully  complete  training  would  also  be  kept 
out;  raising  prerequisite  scores  would  entail  the  cost  of  reducing  the 
eligible  manpower  pool  ar.d  produce  a  small  reduction  in  the  attrition 
rates . 


FAILURE  RATES  AT  VARIOUS  LEVELS  OF  APTITUDE  AREA  SCORES 
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CONCLUSIONS 


!  Test  scores  provide  critical  data  for  making  decisions  about  the  suc¬ 
cess  or  failure  of  trainees.  This  research  focused  on  the  meaning  of 
scores  on  achievement  tests  given  during  the  course,  especially  as  they 
impinge  on  decisions  to  fall  trainees.  The  training  process  itself  was 
not  analyzed  to  determine  the  relationship  between  training  and  achieve¬ 
ment  measures,  nor  was  any  attempt  made  to  determine  absolute  pass-fail 
standards.  The  tests  were  assumed  to  provide  meaningful  measures  of  the 
examinees'  skills  and  knowledge,  and  the  passing  score  as  set  by  the 
school  was  accepted  as  the  standard  determining  pass-fail.  In  subsequent 
research,  the  relationship  between  training  grades  and  performance  on  the 
Job  will  be  analyzed.  The  appropriateness  of  passing  standards  can  then 
be  examined. 

i 

What  is  the  primary  determinant  of  failure  in  these  courses?  Failure 
generally  hinges  cm  achievement  in  the  written  tests  administered  at  the 
end  of  the  reporting  periods.  Failure  rate  on  these  tests  was  generally 
well  over  10  percent--and  occasionally  over  50  percent--of  all  men  who 
completed  the  course.  On  the  performance  tests,  by  contrast,  trainees 
scored  higher.  On  only  two  performance  tests,  both  in  the  Machinist 
course,  did  failure  rates  exceed  20  percent. 

The  ACB  tests  could  identify  some  of  the  failures,  but  raising  the 
prerequisite  scores  would  have  small  impact  on  the  failure  rates.  These 
results  suggest  that  the  most  fruitful  strategy  would  be  a  thorough 
analysis  of  the  written  achievement  tests  to  determine  whether  the  minimum 
passing  scores  are  realistic  and  to  judge  the  correspondence  between  train¬ 
ing  materials  and  test  content. 

One  possibility  is  to  construct  mastery  achievement  tests  which  would 
include  tasks  that  all  trainees  should  have  mastered.  The  failures  would 
then  be  trainees  who  do  not  have  competence  to  perform  at  the  minimal 
standard.  Some  difficult  items  could  also  be  included  to  differentiate 
the  most  able  trainees--for  example,  to  pick  out  the  top  men.  While  some 
bright  students  may  be  motivated  by  difficult  tests,  slow  and  average 
students  are  more  likely  to  be  motivated  by  a  feeling  of  success.  If  the 
written  tests  are  designed  to  assess  minimal  -acceptable  standards,  then 
failure  rates  will  be  lower  and  more  men  will  have  the  experience  of  being 
successf  j1 . 

The  low  degree  of  relationship  between  written  and  performance  tests, 
both  in  terms  of  correlation  and  percentage  of  failures  on  each,  raises 
questions  about  the  meaning  of  the  measures.  Both  types  of  test  are 
estimates  of  the  skills  and  knowledge  covered  during  a  particular  report¬ 
ing  period.  What  are  the  conditions  that  lead  to  a  trainee's  doing  well 
on  one  type  but  not  on  the  other?  An  obvious  explanation  is  that  some 
msn  are  good  with  practical,  hands-on  learning,  while  others  are  better 
with  theory  and  concepts.  Yet  the  lack  of  consistency  in  scores  on  the 
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performance  test,  at  shown  by  their  low  intercorrelations,  raises  ques¬ 
tions  about  their  meaning.  Performance  .tests  were  highly  correlateo 
with  final  course  grades,  but  these  were  part-whole  relationships,  with 
the  performance  tests  heavily  weighted.  Evidence  seems  to  indicate  thot 
performance  tests  need  improvement  as  evaluation  tools. 

Performance  tests  are  receiving  increasing  emphasis  in  Army  MOS  train¬ 
ing  and  in  education  in  general.  Their  Increased  significance  and  low 
intercorrelation  suggest  that  further  research  to  Improve  their  utility  is 
warranted.  Two  immediate  questions  concern  reliability  of  measurement: 
l)  How  consistent  are  trainees  in  performing  the  same  or  similar  tasks 
on  separate  occasions?  2)  How  consistent  are  the  same  or  different  ob¬ 
servers  in  scoring  trainee  performance?  Another  question  concerns  the 
overlap  between  the  tasks  in  performance  tests  and  the  training  objectives: 
Do  the  tasks  in  performance  tests  adequately  sample  the  range  of  skills 
and  knowledge  and  can  meaningful  pass-fail  distinctions  be  made?  Since 
the  decisions  about  trainees'  scores  on  performance  tests  play  such  a 
significant  role  in  evaluating  their  progress,  a  thorough  understanding 
of  the  virtues  and  pitfalls  of  performance  tests  would  help  guide  the 
instructional  staff. 

The  effort  to  improve  performance  tests  should  be  well  worth  the  cost 
since  they  have  such  high  face  validity.  Even  as  these  tests  now  stand, 
the  staff  of  the  Ordnance  School  report  that  performance  tests  serve  as 
good  teaching  devices  by  helping  the  trainee  solidify  what  he  has  been 
taught.  The  research  on  performance  tests  could  be  directed  to  improve 
them  as  evaluation  devices. 

The  ACB  tests  had  higher  validity  for  predicting  grades  on  written 
tests  than  on  performance  tests.  One  reason  is  that  both  the  aptitude 
and  written  tests  are  paper-and-pencil  instruments  that  tap  cognitive 
rather  than  motor  skills.  The  question  of  reliability  of  performance 
tests,  though,  must  be  raised  again  in  the  context.  If  performance  tests 
have  low  reliability,  they  cannot  be  predicted  by  any  type  of  measure, 
including  other  performance  tests.  The  lower  intercorrelations  among 
performance  tests  supports  the  hypothesis  that  performance  tests  are  not 
even  predictable  by  similar  measures.  From  an  instructional  point  of  view, 
performance  tests  are  excellent  measures,  because,  after  all,  the  objec¬ 
tives  of  training  in  these  courses  are  to  teach  men  to  manipulate  equip¬ 
ment.  From  a  measurement  point  of  view,  however,  decisions  about  the 
evaluation  to  place  on  trainee  performance  may  not  be  as  accurate  as  is 
expected  from  paper-and-pencil  tests. 

The  next  phase  of  this  research  project  is  to  examine  the  relation¬ 
ships  among  aptitude  te:ts,  training  grades,  and  performance  on  job  tasks. 
Perhaps  these  analyses  will  increase  our  understanding  of  how  written  and 
performance  tests  function  in  the  total  process  of  classifying,  training, 
and  utilizing  Army  enlisted  men. 
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An  important  function  of  achievement  tests  is  to  determine  which 
students  have  developed  an  adequate  level  of  skills  and  knowledge. 
Achievement  tests  are  also  used  to  make  distinctions  among  those  passing, 
for  example,  to  assign  grades  of  A,  B,  or  C  or  to  identify  the  top 
students  in  a  class.  The  measurement  instruments  used  are  critical  in 
assessing  the  students.  The  tests  must  be  relevant  to  the  material 
taught  (a  matter  of  expert  judgment)  and  they  must  make  adequate  and 
reasonable  discriminations  among  different  levels  of  achievement.  To 
make  an  intelligent  judgment  about  the  adequacy  and  reasonableness  of 
tests  applied  in  a  course,  an  analysis  of  the  scores  is  required.  In 
the  present  phase  of  the  research,  the  achievement  test  scores--written 
and  performance--obtained  at  the  end  of  each  reporting  period  in  three 
job  training  courses  were  analyzed  in  relation  to  each  other,  to  final 
course  grade,  and  to  aptitude  test  scores  obtained  prior  to  training. 

The  research  was  designed  to  provide  data  about  the  ways  the  tests  are 
actually  functioning.  The  results  can  be  evaluated  in  the  context  of 
the  instructional  model  on  which  the  courses  are  based.  The  question  is 
whether  the  results  are  in  keeping  with  the  instructional  model  or  there 
is  some  dissonance  between  observed  and  expected  outcomes. 
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GRADING  PLAN  FOR  SMALL  ARMS  MECHANIC  COURSE  (45B) 


Totals  280  100  10.0  26.0  64.0 


*Sjmpl«  a  coni  lets  of  all  man  who  ttartad  Couria  4  4  E  (N  ■  111) 

6 Sample  B  comliti  only  of  man  who  complatad  Couria  44E  (N  « 113) 


Table  B-2 


MEANS  AND  STANDARD  DEVIATIONS  OF  VARIABIES  IN  TWO  SAMPLES 
FOR  SMALL  ARMS  REPAIR  COURSE  (45B) 


Variables 

Mean 

A*  Bb 

Std.  Dev. 

A  B 

ACB  Test 


VE 

111.9 

112.9 

20.9 

20.6 

AR 

106.8 

108.3 

20.3 

19.0 

SM 

111.0 

111.6 

14.2 

14.1 

PA 

110.6 

111.5 

19.5 

19-1 

ACS 

108.2 

109.1 

16. 9 

16. 5 

AI 

105.1 

106.0 

16. 7 

16.4 

MA 

IO9.O 

110.4 

18.6 

17-4 

ELI 

108.0 

109.1 

19.4 

19.1 

GIT 

106. 3 

107-3 

18.8 

13.3 

Cl 

102.9 

103-9 

22.4 

22.3 

ARC 

98.5 

99-6 

28.1 

27.8 

Written  Test 

1 

74.5 

75-3 

14.4 

14.1 

2 

76.4 

77-3 

15.1 

14.6 

3 

77-2 

77-4 

14.3 

14.4 

4 

74.6 

73-3 

14.7 

14.3 

5 

71.0 

71-9 

15-5 

15- c 

6 

81.2 

81.9 

12.7 

12.1 

7 

85.I 

85.4 

11.6 

11.5 

8 

76.1 

76-9 

13.5 

12.9 

9 

66.2 

66.7 

11.6 

11.4 

Performance  Test 

1 

84.2 

84.6 

10.2 

9-9 

2 

82.1 

82.1 

11.2 

11.2 

6 

80.2 

80.5 

12.7 

12.4 

9 

78.9 

79-5 

7-8 

6.8 

Final  Course 

Grade 

77-0 

78.3 

6.7 

6.0 

*Sampl«  A  consltti  of  oil  mon  who  it  irtod  CourM  4  $  B  (N  1 3 1 2 ) 
bSampIo  B  conilit*  only  of  mon  who  complotod  Courto  4SB  (N  i  292 ) 
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Table  B-3 


MEANS  AND  STANDARD  DEVIATIONS  OF  VARIABLES  IN  TWO  SAMPLES 
FOR  FUEL  AND  ELECTRICAL  SYSTEM  REPAIR  COURSE  (63G-01d) 


Variables 

Mean 

A*  B> 

Std.  Dev. 

A  B 

ACB  Test 


VE 

103.4 

104.6 

18.4 

18.0 

AR 

100.9 

101.9 

18.3 

18.2 

SM 

108.2 

109.3 

16.1 

15.5 

PA 

101.2 

102.2 

20.5 

20.3 

ACS 

107.2 

108.0 

15.9 

15.6 

AI 

114.4 

115.5 

14.5 

13.5 

MA 

107.1 

108.1 

15.1 

14.3 

ELI 

107.9 

108.6 

18.1 

17.8 

GIT 

101.3 

102.2 

15.5 

14.9 

Cl 

95.9 

96.5 

17.4 

16. 9 

ARC 

97.2 

98.1 

27.2 

27.1 

Written  Test 

1 

87.6 

88.2 

9.1 

8.7 

2 

86.2 

87.3 

12.2 

11.1 

3 

76.5 

77.2 

12.8 

12.4 

4 

75.3 

75-5 

14.7 

14.8 

5 

72.5 

72.7 

I6.3 

16.4 

6 

78.5 

78.8 

12.1 

11.9 

7 

69.6 

69.6 

15.2 

15.2 

8 

79.6 

79.6 

11.3 

11.3 

9 

75-7 

75-7 

12.3 

12.3 

10 

78.0 

78.0 

11.1 

11.1 

11 

80.5 

80.4 

15.4 

15.4 

12 

67*3 

67-3 

10.8 

10.8 

Performance  Test 

8 

94.1 

94.I 

6.8 

6.8 

11 

95-0 

95  0 

7-3 

7-3 

12 

93.3 

93.3 

5.0 

5-0 

Final  Course 

Grade 

81.1 

82.1 

6.8 

5-5 

(Sampla  A  con  lift!  of  all  man  who  started  Court#  63Q-OIO  (N  s  212) 
°S#mpla  B  conilsti  only  of  man  who  complatad  Coursa  S3Q>Old  (N  :  200) 
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Table  B-4 


MEANS  AND  STANDARD  DEVIATIONS  OF  VARIABLES  IN  TOO  SAMPLES 
FOR  FUEL  AND  ELECTRICAL  SYSTEM  REPAIR  COURSE  (63G-New) 


Variables 

Mean 

A*  Bb 

Std. 

A 

Dev. 

B 

ACB  Test 

VE 

99-3 

100.2 

18.7 

18.2 

AR 

96.  Q 

100.0 

18.4 

17.9 

SM 

104.6 

104.6 

16.1 

16.0 

PA 

98.7 

99-3 

21.7 

21.3 

ACS 

103.3 

103.4 

18.5 

18.4 

AI 

112.1 

112.6 

15.0 

14. q 

MA 

105.1 

105.7 

15.3 

15.2 

ELI 

103.3 

104.1 

17.3 

17.2 

GIT 

9S.3 

99.1 

16.0 

15.8 

Cl 

92.2 

92.3 

20.4 

20.7 

ARC 

92.4 

93-4 

28.3 

28.5 

Written  Test 

1 

71.0 

72.1 

14.2 

13.3 

2 

71.6 

72.7 

15.2 

14.0 

3 

71.2 

71.4 

15.9 

15.9 

4 

73.9 

74.0 

15.1 

15.1 

5 

72.1 

72.2 

14.2 

14.2 

6 

62.0 

62.0 

10.8 

10.8 

7 

57-3 

57-3 

13.2 

13.2 

8 

55-8 

55-8 

12.2 

12.2 

Performance  Test 

1 

90  8 

91.0 

6.5 

6.5 

3 

96.8 

96.9 

4.4 

4.4 

5 

94.1 

94.1 

5.5 

3.5 

6 

89.3 

89.4 

10.4 

10.4 

7 

94.6 

94.6 

5.2 

5.2 

8 

93.6 

93-6 

3-8 

3.8 

Final  Course 

Grade 

81.9 

83.4 

6.6 

4.4 

A  contltu  of  all  man  who  itartad  Couria  (tlQ-Naw  (N  s  204) 
^Sampla  Bcontlits  only  of  man  who  complatad  Couria  630-Naw  (N  s  117) 


APPENDIX  C  •  CORRELmTION  COEFFICIENTS  FOR  COURSES 

Table  C-l 

CORRELATION  COEFFICIENTS  FOR  MACHINIST  COURSE  ( 44E -EARLY) 


Written 

Tests 

Mean  Correlation 

Perfor¬ 

mance 

Tests 

Final 

Course 

Test 

Written  Test 

1 

.  64 

.48 

•  73 

2 

•  35 

.28 

.40 

3 

.62 

.48 

•71 

4 

.64 

.48 

•  73 

5 

.62 

.50 

•76 

6 

.58 

•47 

.71 

7 

.60 

.41 

.64 

8 

.60 

.41 

.66 

9 

•55 

.40 

.63 

Performance  Test 

2 

.45 

.46 

.69 

3 

•  36 

•  30 

.46 

4 

.50 

.48 

.72 

7 

.36 

.34 

.56 

8 

.47 

.47 

.78 

9 

•  47 

.48 

.88 

ACB  Test 

VE 

•53 

•30 

.48 

AR 

•  53 

•  33 

.49 

SM 

.45 

•59 

•55 

PA 

•51 

.44 

.62 

ACS 

•  30 

.20 

•  27 

AI 

.44 

.36 

.48 

MA 

•  54 

.44 

.62 

ELI 

•45 

•  34 

•50 

GIT 

.50 

•  30 

•47 

Cl 

•  33 

CO 

CVJ 

•  36 

ARC 

•39 

.29 

.42 

Table  C-2 


CORRELATION  COEFFICIENTS  FOR  MACHINIST  COURSE 


Mean  Correlation 
Per for - 

Wrltten  mance 

Tests  1  Tests 


( 44E- LATE) 


Final 

Cturse 

Gride 


Table  C-3 

CORRELATION  COEFFICIENTS  FOR  SMALL  ARMS  REPAIR  COURSE  (45B) 


1 


Mean  Correlation 


Perfor-  Final 

Written  mance  Course 

Tests  Tests  Grade 


Table  C-4. 


CORRELATION  COEFFICIENTS  FOR  FUEL  AND  ELECTRICAL 
SYSTEM  REPAIR  COURSE  (63G-OLD) 


Mean  correlation 


Perfor-  Final 

Written  mance  Course 

Tests  Tests  Grade 


Written  Test 


1 

•57 

•  33 

•  71 

2 

•57 

•  31 

.72 

3 

.66 

•  43 

.85 

4 

•59 

•  39 

•  76 

5 

.58 

.40 

•  76 

6 

.62 

•  36 

.79 

7 

•57 

•  39 

•  73 

8 

.68 

•47 

.87 

9 

.61 

.40 

.80 

10 

.61 

.40 

•  79 

11 

•  54 

•59 

.72 

12 

.69 

•  47 

.91 

Performance  Test 

8 

.48 

.48 

•  65 

11 

.23 

.31 

•  36 

12 

•47 

.45 

.66 

ACB  Test 

VE 

•  54 

•  23 

.64 

AR 

.58 

•  35 

•  71 

SM 

.48 

•53 

.61 

PA 

.45 

•25 

•55 

ACS 

•51 

.26 

•  39 

AI 

•52 

.41 

.68 

MA 

•55 

.41 

•  71 

ELI 

•  52 

.28 

.64 

GIT 

•57 

•33 

•70 

Cl 

•  29 

.21 

•  37 

ARC 

•  56 

.22 

.44 
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Table  C-5 


CORRELATION  COEFFICIENTS  FOR  FUEL  AND  ELECTRICAL 
SYSTEM  REPAIR  COURSE  ( 63G-NEW) 


Mean  Correlation  with  Training  Performance 


Written 

Tests 

Perfor¬ 

mance 

Teats 

Final 

Course 

Grade 

Written  Test 

1 

.68 

•  59 

.83 

2 

.61 

•  31 

•  73 

3 

.65 

.40 

.81 

4 

.54 

•  30 

.67 

5 

.68 

.38 

.82 

6 

.67 

•  39 

.82 

7 

.65 

•  38 

.80 

8 

•  70 

.41 

.87 

Performance  Test 

1 

.09 

.09 

.17 

3 

.26 

.20 

.39 

5 

•51 

.33 

.70 

6 

•  38 

.27 

.62 

7 

.37 

.28 

.54 

8 

.60 

.40 

.80 

ACB  Test 

VE 

.50 

•  29 

.60 

AR 

•55 

■35 

•  67 

SM 

•47 

•25 

•55 

PA 

.42 

•  29 

•53 

ACS 

.27 

.23 

•  36 

AI 

•51 

•  23 

•57 

MA 

.  46 

.26 

•54 

ELI 

.48 

.23 

•54 

GIT 

.50 

•  29 

.60 

Cl 

.21 

.12 

.27 

ARC 

•29 

.18 

•35 

APPENDIX  D 


PREDICTING  FINAL  COURSE  GRADE  FROM 
GRADES  IN  REPORTING  PERIODS 


An  Important  question  from  the  Instructional  staff  point  of  view  Is 
how  early  In  training  courses  the  eventual  failures  can  be  Identified. 

If  the  failures  can  be  Identified  early,  then  they  can  be  sent  to  other 
courses  where  they  are  more  likely  to  succeed.  Both  the  man  and  school 
would  benefit  from  early  transfer  before  both  have  a  large  investment  In 
a  course  where  the  likely  outcome  is  a  failure. 

The  identification  of  early  failures  was  examined  via  the  correlation 
between  scores  on  written  and  performance  tests  given  at  the  end  of  report¬ 
ing  periods  and  final  course  grades.  The  sums  of  the  examination  grades 
through  each  reporting  period  were  correlated  with  final  course  grade, 
and  the  results  are  shown  in  Table  D-l.  The  first  row  shows  the  corre¬ 
lation  coefficients  of  the  examination  of  reporting  period  1  with  final 
course  grade.  The  second  row  shows  the  correlation  of  the  sum  of  exami¬ 
nation  scores  of  reporting  periods  1  and  2.  In  subsequent  rows,  the 
scores  for  additional  reporting  periods  were  added,  until,  in  the  final 
row,  the  examination  scores  for  all  reporting  periods  were  added  togeth¬ 
er.  Correlation  coefficients  began  at  about  .6  for  period  1  and  In¬ 
creased  to  about  .9  or  above  for  the  sum  of  all  scores  through  the  final 
period. 

Table  D-l 

Correlation  of  Sum  of  Examinations 
with  Final  Course  Grade 


Reporting  Course  Examinations 

Periods _ 44E  45B  65G  New 


1 

.60 

.67 

.66 

1+2 

.65 

•  77 

.72 

1+2+3 

.70 

•  76 

.81 

1-4 

.80 

•  76 

.82 

1-5 

.81 

•79 

.87 

1-6 

.82 

.85 

•92 

1-7 

.85 

.86 

.94 

1-8 

.88 

.86 

•95 

1-9 

•  94 

.89 

Preceding  page  blank 
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The  following  chart  can  be  used  to  interpret  the  meaning  of  the  cor¬ 
relation  coefficients  for  identifying  men  who  will  fail  the  course.  Four 
levels  of  correlation  between  examination  scores  and  final  course  grade 
are  shown.  Three  levels  of  failure  on  examinations  are  also  shown:  7%, 
fairly  easy  examinations;  16$, moderately  difficult;  and  50$,  a  difficult 
examination.  The  failure  rate  for  the  course  was  set  at  16$,  which  is 
about  the  average  for  the  three  courses  in  this  research.  The  cell  en¬ 
tries  show  the  percentage  of  men  expected  to  fail  both  the  examination 
and  the  course. 


$  that  fail  examination 

7$  16$  50$ 

correlation  .60 
with  final  .70 
course  grade. 80 

.90 


The  first  cell  entry,  4$,  means  that  of  the  7$  that  fail  the  examination, 
4$  would  also  eventually  fail  the  course,  and  the  remaining  5$  would 
improve  their  grades  and  pass  the  course.  An  additional  12$  who  passed 
this  examination  would  fail  subsequent  examinations  and  eventually  the 
course.  As  the  correlation  increases,  the  accuracy  of  prediction  in¬ 
creases.  As  the  percentage  of  failures  on  an  examination  increases,  so 
does  the  proportion  of  those  who  fail  the  course.  When  the  failure  rate 
on  the  examination  is  30$  and  correlation  with  final  course  grade  is 
r“.90,  then  15$  of  the  eventual  failures  would  be  identified;  only  1$ 
of  the  course  failures  would  pass  the  early  examination.  But,  of  course, 
another  15$  that  failed  the  examination  completed  the  course  successfully. 

The  chart  can  be  used  to  estimate  the  proportion  of  course  failures. 
The  data  required  are  the  correlation  of  the  sum  of  the  examination 
scores  with  final  course  grade  and  the  percentage  that  has  failed  the 
examinations;  the  failure  rates  for  the  separate  examinations  will  need 
to  be  averaged  if  more  than  one  is  included.  If  the  failure  rate  for  a 
course  differs  somewhat  from  16$,  say,  plus  or  minus  10$,  then  the  cell 
entries  can  be  changed  proportionally;  or  if  the  failure  rates  on  exami¬ 
nations  fall  between  7$  and  50$,  the  cell  entries  can  be  used  to  estimate 
the  proportion  that  fail  both  the  examinations  and  course.  Since  the 
relationships  .are  not  linear,  extreme  deviations  are  not  proportional 
to  the  tabled  entries.  A  new  table  would  need  to  be  prepared  from  the 
normal  bivariate  tables  for  different  failure  rates. 


-38- 


